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Abstract 

The  purpose  of  this  paper  is  to  present  a  methodology  for  the  design  of  experiments  to  examine 
organizational  behavior  of  command  and  control  teams  operating  in  a  complex  mission 
environment.  This  methodology  can  be  used  to  design  an  executable  model  for  human-in-the-loop 
model-based  experiments  that  provide  the  necessary  empirical  components  for  current  and  future 
research  in  adaptive  C2  architectures. 

1.  Motivation 

Today’s  revolution  in  information  technology  is  changing  the  way  people  within  military 
organizations  interact  and  deal  with  conflicts.  Indeed,  the  changing  patterns  of  potential  threats 
and  conflicts  in  today’s  world  require  a  strong  military  capability  that  is  sufficiently  versatile  to 
execute  a  national  strategy  across  a  full  range  of  operations  -  to  include  war  and  operations  other 
than  war.  Advanced  information  systems  and  capabilities  must  be  tailored  to  support  new  military 
Command  and  Control  (C2)  roles  and  requirements.  In  view  of  such  demands  on  modern  C2 
decision-making  systems,  experimentation  is  put  forth  as  a  way  to  examine  human  decision¬ 
making  and  coordination  processes  and  to  identify  organizational  structures  that  lead  to  superior 
performance. 

While  drawing  inferences  from  empirical  data  is  an  important  element  in  gaining  scientific 
knowledge,  in  many  cases  the  large  cost  associated  with  experimentation  makes  it  impractical  to 
rely  on  inductive  reasoning  only.  This  is  especially  true  when  dealing  with  organizational  behavior 
of  human  teams  designed  to  operate  in  a  complex  multi-task  mission  environment.  The  high  cost 
of  experimentation  demands  a  skillful  experiment  design.  Furthermore,  in  order  to  formulate  the 
hypotheses  to  be  tested  empirically,  one  needs  to  identify  (as  well  as  understand)  the  key  variables 
of  the  underlying  system  structure.  Thus,  to  design  an  experiment,  a  normative  model  of  the 
process  is  needed  to  formulate  the  hypotheses,  to  devise  organizational  structures,  and  to 
develop  a  set  of  measures  to  characterize  various  dimensions  of  organizational  performance.  The 
challenge  facing  the  scientific  community  is  to  develop  and  empirically  validate  theories  and 
models  of  human  decision-making  in  distributed  systems  -  models  that  could  ultimately  contribute 
to  design  modifications  that  enhance  the  overall  human-machine  system  performance. 
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2.  Organizational  Design  Problem 


Over  the  years,  research  in  organizational  performance  has  demonstrated  that  a  strong 
relationship  exists  between  the  specific  structure  of  a  task  environment  (i.e.,  mission)  and  the 
concomitant  organizational  design.  Subsequently,  the  optimality  of  an  organizational  design 
depends  on  the  actual  mission  parameters  (and  organizational  constraints)  -  i.e.,  there  is  no 
universally  “best”  organization.  This  premise  has  led  to  the  application  of  systems  engineering 
techniques  to  the  design  of  human  organizations. 

The  systems  engineering  approach  to  organizational  design  is  as  follows.  First,  a  quantitative 
model  describing  the  mission  and  the  organizational  constraints  is  built.  Then,  one  or  more 
objective  functions  for  the  design  are  specified,  and  an  organization  is  designed  to  optimize  the 
objective  function(s).  When  the  objective  function  includes  several  non-commensurate  criteria, 
the  organizational  design  problem  is  treated  as  a  multi-objective  optimization  problem. 

3.  Modeling  Paradigm 

The  power  of  quantitative  modeling  lies  in  its  ability  to  describe  the  structure  of  a  mission  and  of 
an  organization  by  a  relatively  limited  set  of  fundamental  elements  (or  entities),  parameters, 
variables,  laws,  and  principles.  The  last  two  specify  the  functional  interdependencies  among  the 
structural  elements  and  the  dynamics  of  system  parameters  and  variables. 

It  is  easier  to  establish  functional  reciprocity  between  the  structures  of  a  mission  and  of  an 
organization  if  both  are  modeled  in  a  unified  manner.  The  sine  qua  non  duality  in  modeling  a 
mission  and  an  organization  can  be  achieved,  for  example,  by  employing  graph  formalism  to 
describe  both  structures.  The  graph  formalism  has  the  advantage  of  representing  the  internal 
system  dependencies  in  a  concise,  illustrative,  and  numerically  efficient  way.  In  addition,  use  of 
color-coding  allows  the  graphical  model  to  reveal  more  than  just  quantitative  aspects  of  a  system. 

When  modeling  a  mission  or  an  organization,  the  designer  must  take  into  account  the  complexity 
of  the  underlying  problem,  as  well  as  the  computational  resources  available  to  “solve”  the 
mathematical  models.  The  degree  of  granularity  of  a  model  must  be  adjusted  to  allow  for  real¬ 
time  solution  of  the  resulting  design  problem.  In  addition,  different  modeling  techniques  (e.g., 
different  ways  of  decomposing  a  mission  into  its  constituent  tasks)  will  generally  result  in  different 
mission  models,  and  ultimately  may  lead  to  different  organizational  designs.  Thus,  adopting  the 
“right”  models  of  a  mission  and  an  organization  is  the  first  essential  step  toward  designing  truly 
optimal  organizational  structures. 

3.1  Representing  Mission  and  Organization 

3.1.1  Modeling  a  Mission 

Over  the  years,  research  in  team  decision-making  has  demonstrated  that  an  organization  operates 
best  when  its  command  and  control  architecture  -  its  organizational  structure  and  processes  -  fits, 
or  matches,  the  mission  task  environment.  Consequently,  it  has  been  concluded  that  the 


optimality  of  an  organizational  design  ultimately  depends  on  the  actual  mission  structure.  Hence, 
devising  an  analytical  model  of  the  mission  is  the  first  step  toward  designing  an  optimal  C2 
organization. 

Devising  an  analytical  model  of  the  mission  implies  the  identification  of  a  set  of  fundamental 
mission  elements  (entities),  the  functional  interdependencies  of  their  attributes,  and  principles 
governing  the  evolution  of  the  attributes  over  time.  A  fundamental  question  underlying  a 
distributed  organizational  design  -  ‘  who  should  do  which  part  of  the  missionT  -  implies  that  the 
mission  must  be  decomposable  into  a  set  of  tasks.  Although  a  task  constitutes  an  essential 
mission  entity,  elements  that  determine  the  structure  of  the  mission  are,  in  general,  not  limited 
exclusively  to  tasks.  Mission  entities  can  stipulate  the  natural  domain,  as  well  as  the  external 
objects,  conditions,  and  processes  that  influence  the  dynamics  of  the  mission  (e.g.,  terrain, 
weather,  road  network,  population  areas,  enemy  platforms,  neutral  objects,  and  so  on). 

Every  task  in  itself  represents  a  “small  mission”,  and  can  oftentimes  be  further  decomposed  into 
more  elementary  tasks,  thus  increasing  the  number  of  mission  entities  and,  consequently,  the 
complexity  of  the  model.  The  number  of  mission  entities  determines  the  granularity  of  the  model; 
it  must  be  adjusted  to  allow  for  an  efficient  computational  solution  of  the  resulting  design 
problem. 

The  identification  of  mission  entities  and  their  attributes  can  be  derived  from  a  verbal  description 
of  a  mission.  In  addition,  a  variety  of  modeling  techniques  (e.g.,  Object  Oriented  Modeling 
(OOM);  Entity  Relationship  (ER)  [Elmasri  et  al.,  1994];  Integrated  Definition  (IDEF)  Modeling 
[NIST,  1993a  and  1993b];  Relational  Database  Design  (RDD))  can  be  applied  to  capture  the 
internal  structure  of  a  mission.  One  of  the  most  popular  modeling  methodologies  employs  a 
graph  formalism  to  describe  the  mission  structure.  The  graph  formalism  may  be  used  to  construct 
various  dependency  diagrams  to  specify  different  dimensions  of  the  mission  structure  and  its 
evolution  over  time  (e.g.,  an  entity  relationship  diagram,  a  task  precedence  diagram,  a  data-flow 
diagram,  and  so  on). 

We  conclude  that  devising  an  analytic  model  of  a  mission  requires  the  identification  of:  (i)  entity 
types',  (ii)  attribute  types',  and  (iii)  relationship  types.  To  identify  the  entity  types,  different 
mission  decomposition  techniques  can  be  applied.  After  distinguishing  a  set  of  tasks  and  objects 
that  represent  mission  entities,  a  modeler  specifies  their  attribute  values,  and  formulates 
quantitative  interrelationships  among  the  attributes  via  equations.  To  complete  the  analytic  model 
of  a  mission,  various  dependency  diagrams  are  constructed  to  delineate  the  principles  governing 
the  joint  evolution  of  attributes  over  time. 

While  many  different  models  of  the  mission  can  be  constructed,  it  is  important  to  select  the  “right” 
model  that  highlights  those  dimensions  of  the  mission  structure  that  are  the  most  relevant  to  the 
prospective  organizational  design  process.  Thus,  a  comprehensive  mission  modeling  process 
consists  of  the  following  steps: 

•  choosing  an  appropriate  mission  task  decomposition(s)  and  identifying  the 
mission  tasks; 

•  specifying  the  task  attributes; 


•  representing  functional  dependencies  among  the  task  attributes  via  equations; 

•  constructing  the  task  dependency  diagrams. 

3.1.2  Defining  Organization 

One  of  the  goals  in  studying  C2  team  decision-making  is  the  development  and  subsequent 
validation  of  theories  and  models  that  could  ultimately  suggest  organizational  design 
modifications  to  enhance  the  overall  human-machine  system  performance.  The  application  of 
systems  engineering  techniques  to  modeling  and  design  of  organizations  allows  one  to  establish  a 
mathematical  framework  for  optimizing  predicted  human  team  performance. 

The  problem  scope  and  complexity  faced  by  large-scale  C2  systems  that  involve  humans, 
machines,  computers,  networks,  and  databases  interacting  within  an  organization  often  require 
that  the  decision-making  and  operational  functions  be  distributed  over  several  human  subjects. 
Thus,  a  characteristic  feature  of  such  systems  is  the  presence  of  teams  of  human  decision-makers 
(DMs),  who,  while  geographically  separated,  must  coordinate  their  information,  resources,  and 
activities  in  order  to  achieve  their  common  goal  in  what  is  generally  a  complex,  dynamic,  and 
uncertain  mission  environment. 

Since  the  decision-making  and  operational  capabilities  of  a  human  are  limited,  the  distribution  of 
information,  resources,  and  activities  among  DMs  must  be  set  up  accordingly,  with  the  decision¬ 
making  and  operational  load  of  each  DM  remaining  below  the  corresponding  thresholds.  For 
example,  when  the  overall  information  available  to  a  team  during  the  mission  overwhelms  the 
information  processing  capabilities  of  an  individual  DM,  the  team’s  information  acquisition  and 
decision-making  must  be  decentralized  so  that  operators  can  focus  on  their  local/relevant 
information. 

As  a  consequence  of  decentralization  in  large-scale  systems,  each  DM  only  has  access  to  a  portion 
of  the  information  available  to  the  team.  Moreover,  in  realistic  situations,  the  total  information  set 
may  be  incomplete  and  inaccurate  due  to  lax  updating,  missed  detection  of  events,  information 
whose  quality  changes  with  time,  and  errors  in  data  taking.  Hence,  team  information  processing 
is  characterized  by  a  high  level  of  uncertainty  with  partial  overlap  of  information  among  DMs. 
The  critical  issues  in  team  information  processing  are:  who  should  know  what,  who  should 
communicate  what  and  with  whom,  and  when  people  should  and  should  not  communicate? 

The  total  decision-making  and  operational  load  is  generally  partitioned  among  DMs  by 
decomposing  a  mission  into  tasks  and  assigning  these  tasks  to  individual  decision-makers  who  are 
responsible  for  their  planning  and  execution.  Moreover,  an  overlap  in  task  processing  (wherein 
two  or  more  DMs  share  responsibility  for  a  given  function/task  while  each  possesses  the  capability 
to  process  the  task)  gives  the  team  a  degree  of  freedom  to  adapt  to  uneven  demand  by 
redistributing  load.  The  critical  issues  in  team  task  processing  are:  what  should  be  done,  who 
should  do  what,  and  when ? 

In  general,  decision-makers  are  provided  with  limited  resources  with  which  to  accomplish  their 
objectives  either  in  information  processing  or  in  task  processing.  The  distribution  of  these 
resources  among  DMs  and  the  assignment  of  these  resources  to  seek  information  and  to  process 


tasks  are  key  elements  in  an  organization’s  design.  Team  members  must  dynamically  coordinate 
their  resources  to  process  their  individual  tasks  while  assuring  that  team  performance  goals  are 
met.  The  critical  issues  in  team  resource  allocation  are:  who  should  own  or  transfer  a  specific 
resource,  when ,  and/or  how  long ? 

The  tentative  classification  of  a  DM’s  mission  processing  activities  into  decision-making  and 
operations  can  be  further  expanded,  and  the  corresponding  thresholds  can  be  estimated.  For 
example,  activities  such  as  communication,  target  detection,  external  information  release,  platform 
maneuvering,  firing  weapons,  and  so  on,  -  all  represent  various  dimensions  of  a  DM’s  load.  The 
interdependencies  among  the  levels  of  the  thresholds  corresponding  to  these  dimensions  of  a 
DM’s  workload  delineates  one  of  the  key  principles  describing  the  behavior  of  a  human  DM. 
When  a  modeler  needs  to  emphasize  the  individual  differences  among  DMs,  he  can  do  so  by 
specifying  an  expertise  level  of  each  DM  with  respect  to  a  particular  DM  activity.  For  example, 
the  expertise  can  be  specified  by  stating  the  likelihood(s)  that  a  DM  can  solve  "model"  problem(s) 
using  the  necessary  resources  and  maintain  the  required  speed  of  task  processing. 

The  sine  qua  non  duality  in  modeling  a  mission  and  an  organization  can  be  observed  by 
recognizing  that  it  is  impossible  to  classify  all  DM  activities  without  the  knowledge  of  the  type  of 
mission  that  the  organization  will  be  facing.  In  turn,  to  decompose  a  mission  into  tasks  requires 
the  knowledge  of  resources  available  to  an  organization.  Hence,  it  is  important  that  the  modeling 
of  both  the  mission  and  organization  be  carried  out  simultaneously  to  elucidate  the  functional 
reciprocity  between  the  two  structures. 

While  organizational  design  specifies  the  distributed  nature  of  mission  processing  by  assigning 
each  DM  his  share  of  information,  resources,  and  activities,  a  command  hierarchy  among  DMs 
designates  their  control  responsibilities  (through  command  authority)  and  regulates  the  inter-DM 
coordination  (by  assigning  the  responsibility  of  resolving  decision  ambiguities  among  coordinating 
DMs).  In  addition,  organizational  design  can  specify  a  communication  structure  among  DMs 
(e.g.,  a  communication  net)  to  facilitate  distributed  information  processing  required  to  complete 
the  mission. 

In  summary,  when  modeling  an  organization,  we  differentiate  two  classes  of  entities :  (i)  decision¬ 
makers  (DMs)  and  (ii)  resources.  The  key  attributes  in  modeling  a  DM  are  the  individual  DM 
thresholds  with  respect  to  particular  DM  activities  (e.g.,  information  processing  and  operational 
activities).  The  interdependencies  among  the  thresholds  corresponding  to  different  dimensions  of 
a  DM’s  workload  explicates  the  principles  governing  the  joint  evolution  of  DM’s  attributes.  In 
the  model  of  an  organization,  the  classification  of  resource  entities  can  be  further  expanded  (e.g., 
platforms,  machines,  computers,  networks,  databases,  etc.),  with  their  attributes  specified 
accordingly.  Hence,  a  comprehensive  model  of  an  organization  includes: 

•  a  specified  (maximal)  number  of  available  decision-makers  (DMs); 

•  a  classification  of  a  DM’s  activities,  with  a  specification  of  the  corresponding 
thresholds  and  their  inter-relationships; 

•  a  classification  of  resources,  together  with  their  numerical  attributes. 


For  a  given  mission  structure,  including  mission  decomposition  into  constituent  tasks,  and 
organizational  constraints,  a  distributed  organizational  design  must  delineate: 

•  a  coordination  structure  among  DMs; 

•  a  DM-resource  allocation; 

•  a  DM-task  processing  assignment/schedule. 

3.  2  Functionality  of  Organization  as  Hierarchical  Network 

The  advanced  information  systems  and  resource  capabilities  available  to  modern  C2  structures 
will  determine  new  versatile  doctrine  for  organizations  in  their  respective  missions.  One  needs  to 
thoroughly  understand  the  functionality  of  different  components  in  a  C2  structure  to  predict  the 
performance  of  a  particular  organizational  design.  While  the  functions  that  are  carried  out  by  an 
organization  may  vary  based  on  the  makeup  of  a  specific  mission,  the  general  classification  of  the 
basic  processes  inherent  to  a  C2  organization  is  critical  to  the  evolution  of  the  organizational 
design  process.  The  facilitation  of  the  fundamental  processes  (such  as  coordination, 
communication,  management  of  weapons,  operational  planning,  situation  awareness,  dynamic 
distributed  decision-making,  etc.),  common  to  a  large  variety  of  C2  organizations,  is  the  key  to 
superior  organizational  performance. 

We  seek  the  optimal  design  of  adaptive  C2  organizations  to  establish  a  proper  balance  among  the 
following  general  intrinsic  processes: 

•  segmented  information  acquisition  and  processing; 

•  distributed  decision-making; 

•  managing  operational  resources; 

•  inter-DM  action  coordination; 

•  mission  monitoring; 

•  failure  and  anomaly  detection; 

•  strategy  adaptation  /  reconfiguration  enforcement. 

The  classification  of  the  processes  that  define  the  functionality  of  an  organization  provides  a 
modeler  with  a  basis  for  choosing  the  design  parameters  and  performance  measures. 

4.  Optimization  Criteria 

When  speaking  of  design  optimality,  one  should  realize  that  different  people  may  value  different 
criteria  when  judging  the  optimality  of  an  organization,  i.e.,  the  notion  of  optimality  is  subjective. 
In  addition,  in  an  organizational  design  process,  different  objective  functions  often  lead  to 
different  “optimal”  organizational  structures.  Hence,  the  correct  choice  of  optimization  criteria  is 
critical  to  generating  an  optimal  organizational  design.  Relative  weights  of  optimization 
parameters  in  the  objective  function  are  also  important  because  they  impact  the  outcome  of  the 
design  process.  Therefore,  one  of  the  designer’s  goals  is  to  appropriately  select  the  optimization 
criteria  to  adequately  reflect  the  principles  stated  in  the  original  hypotheses. 


5.  Adaptability  Requirements 


When  designing  organizations  to  operate  in  an  uncertain  environment,  the  specific  information 
about  many  mission  parameters  may  be  inaccessible  a  priori,  with  only  estimates  (e.g.,  the  forms 
of  probability  density  functions,  expected  values  and  standard  deviations,  etc.)  available  to  the 
designer.  Once  the  mission  commences,  the  actual  values  of  its  parameters  may  require  the 
implementation  of  a  particular  strategy  to  achieve  desired  performance.  Furthermore,  throughout 
the  course  of  a  mission,  various  events  (e.g.,  an  erroneous  initial  mission  parameter  estimation, 
operational  resource  failures,  malfunctioning  of  a  decision  node,  etc.)  may  trigger  unexpected 
changes  in  either  the  mission  environment  or  in  the  organizational  constraints.  As  a  result  of  such 
changes,  it  may  no  longer  be  possible  for  an  organization  to  successfully  complete  the  mission 
while  maintaining  the  same  strategy  it  was  using  initially.  Moreover,  none  of  the  strategies, 
feasible  for  the  current  organizational  design,  may  allow  the  organization  to  successfully  complete 
the  mission. 

Consequently,  an  on-line  decision  strategy  adaptation  and/or  structural  reconfiguration  could 
become  a  prerequisite  to  the  successful  completion  of  the  mission.  Thus,  the  challenge  is  to 
construct  organizational  architectures  that  can: 

•  maintain  the  mission  schedule ,  while  operating  in  a  time -pressured  and 
uncertain  environment; 

•  capture  the  necessary  information  about  organizational  node  failures  and/or 
unforeseen  dynamic  changes  in  the  mission  environment; 

•  determine  if  an  adaptation  is  required; 

•  analyze  the  available  adaptation  options; 

•  recommend/ adopt  appropriate  forms  of  adaptation  (or  abort  the  mission  in  the 
case  where  no  adaptation  is  available  or  its  cost  is  unacceptably  high); 

•  implement  the  required  on-line  adaptation  of  structure  and/or  strategy. 

For  an  organization  to  be  able  to  adapt  and  still  maintain  its  mission  schedule,  the  adaptation 
phase  must  be  compatible  in  time  with  the  processing  of  mission  tasks  (in  some  cases,  the 
adaptation  time  must  be  significantly  smaller  than  the  time  to  process  the  corresponding  mission 
tasks  to  allow  for  successful  completion  of  the  mission).  Thus,  in  order  to  determine  if  adaptation 
is  required,  the  appropriate  mission/organization  monitoring  data  should  be  analyzed,  and  the 
failure  diagnostics  should  be  performed  in  a  timely  fashion.  For  large  missions,  such  an  analysis 
is  usually  beyond  the  information  processing  capabilities  of  human  organizations.  To  transcend 
the  limitations  of  human  DMs,  appropriate  software  tools  are  required. 

Oftentimes,  it  is  not  efficient  to  analyze  all  available  adaptation  options,  as  such  an  analysis  can  be 
time  consuming  and  computationally  expensive.  When  this  is  the  case,  the  number  of  examined 
options  will  be  restricted  by  the  timeliness  requirements  and  the  processing  speed  of  the 
supporting  algorithms.  Thus,  it  may  be  beneficial  to  look  for  design  solutions  that,  in  the  event 
of  adaptation  triggers  (e.g.,  unexpected  changes  in  the  mission  environment,  resource  failures, 
DM  node  malfunctions,  etc.),  would  allow  for  the  rapid  search  of  adaptation  options  among 
preprocessed  data. 

Since  the  reliability  and  accuracy  of  the  organizational  data,  and  hence  the  ability  of  an 
organization  to  detect  the  need  to  adapt,  are  strongly  dependent  on  the  accuracy  of  the  observed 


measurements,  the  resources  that  accommodate  the  information  acquisition  function  play  a  special 
role  in  the  adaptive  organizational  design.  The  allocation  of  these  resources  to  a  DM  will  impose 
the  additional  monitoring  function/responsibility/workload  on  such  a  DM. 

After  the  proper  adaptation  option  (e.g.,  strategy  shift,  resource  reallocation,  hierarchy 
reconfiguration)  is  selected,  the  organization  needs  to  coordinate  among  its  members  to 
implement  the  selected  change.  In  hierarchical  networks,  the  authority  to  declare  and  enforce  on¬ 
line  reconfiguration  belongs  to  the  superior  DM  (known  as  a  root  DM).  Hence,  it  is  natural 
(although  not  necessary)  that  a  decision  to  adapt  be  taken  at  this  highest  level  of  the  command 
hierarchy. 

6.  Model  Predictions  and  Hypothesis  Testing  Phase 

The  hypotheses  underlying  an  organizational  design  provide  a  terminus  a  quo  for  aggregating  the 
key  model  variables  into  a  set  of  measures  to  characterize  the  organizational  performance.  When 
designing  an  experiment,  the  collected  data  (both  quantitative  and  qualitative)  must  be  sufficiently 
rich  to  advance  the  inductive  inferences  of  the  experiment.  In  the  hypotheses  testing  phase,  a 
numerical  comparison  of  model  predictions  versus  experimental  data  serves  as  a  basis  for  either 
validating  or  rejecting  the  hypotheses.  In  addition,  the  model-data  comparison  helps  to  identify 
human  limitations  and  biases,  and  to  formulate  new  hypotheses  to  be  tested  in  a  Model-Test- 
Model  cycle.  Thus,  the  model  based  quantification  of  hypotheses  and  the  development  of 
quantitative  model  predictions  constitute  critical  steps  in  the  scientific  investigation  of 
organizational  decision-making. 

7.  Setting  Parameters  for  the  Experiment 

In  general,  different  parameters  (e.g.,  coordination,  workload,  resource  allocation)  have  dominant 
effect  on  organizational  performance  under  different  experimental  conditions.  In  order  to  validate 
a  hypothesis  stipulating  the  nature  of  interaction  among  particular  model  variables,  the 
experimental  data  may  prove  ineffective  when  the  variables  of  interest  are  dominated  by  other 
variables  or  measurement  noise.  Therefore,  in  addition  to  quantifying  the  hypotheses  and 
providing  quantitative  performance  measures,  the  experimenter  must  establish  the  relevant 
settings  of  external  parameters  for  the  experiment.  The  search  for  better  parameter  settings 
should  be  based  on  a  fundamental,  model- specified,  understanding  of  the  process  being  studied, 
the  limitations  of  measuring  equipment,  and  the  relationships  the  experimenter  is  explicitly  seeking 
to  determine  or  to  validate. 


8.  Key  Steps  in  Designing  Experiments 


We  conclude  that,  in  order  to  carry  out  a  comprehensive  design  of  experiments  to  examine 
organizational  behavior  of  C2  structures  in  a  complex  mission  environment,  one  needs: 

•  a  modeling  paradigm; 

•  a  method  to  represent  a  mission  and  an  organization; 

•  an  algorithm  to  synthesize  an  organization; 

•  a  procedure  for  measuring  organizational  performance; 

•  a  way  to  evaluate  hypotheses  prior  to  experiment; 

•  a  process  to  establish  relevant  parameter  settings  for  the  experiment. 


The  basic  steps  in  developing  an  executable  model  of  an  experiment  to  test  hypotheses  about  a  C2 
design  are  outlined  in  Fig.l. 
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Figure  1.  Outline  for  analytic  design  of  experiments  in  a  model-test-model  cycle. 

9.  Summary  and  Research  Prospective 

In  this  paper,  we  presented  guidelines  for  designing  experiments  to  examine  organizational 
behavior  of  command  and  control  teams  operating  in  a  complex  mission  environment.  We 
discussed  a  joint  model  of  a  mission  and  an  organization  to  specify  the  structure  of  the  mission, 


organizational  constrains  and  processes,  and  the  requirements  for  an  organizational  design.  Our 
methodology  illustrates  the  underlying  principles  that  guide  the  design  of  organizations. 

This  paper  provides  an  analytical  starting  point  for  model-based  experimental  research  on 
adaptive  architectures.  In  coordination  with  companion  empirical  efforts  (see  [Serfaty,  1996]  and 
[Kemple  et  al.,  1997]),  it  seeks  to  integrate  optimization,  modeling,  and  simulation-based  research 
efforts  with  psychology-based  and  experimental  activities  to  conduct  humans-in-the-loop  model- 
driven  experiments  studying  adaptive  C2  architectures.  The  proposed  models  drive  the 
formulation  of  hypotheses,  the  determination  of  key  variables  and  their  values,  and  prediction  of 
organizational  performance  and  the  processes  of  adaptation.  The  generated  measures  of 
organizational  performance  provide  a  basis  for  post-experimental  model-data  comparison.  In 
addition,  the  implementation  of  the  design  procedure  in  software  allows  one  to  estimate  the 
optimal  experimental  conditions  to  test  a  specific  hypothesis.  The  empirical  findings  will  assist  a 
modeler  in  identifying  new  design  parameters  affecting  organizational  performance  to  further 
study  the  dependency  of  organizational  performance  on  the  mission  structure  and  on 
organizational  design. 
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